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Criteria Preferences in PROMETHEE Methods 
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Abstract: In this paper we extend the PROMETHEE methods to the case of interacting criteria 
on a bipolar scale, introducing the bipolar PROMETHEE method based on the bipolar Choquet 
integral. In order to elicit parameters compatible with preference information provided by the 

u 

. Decision Maker (DM), we propose to apply the Robust Ordinal Regression (ROR). ROR takes 

[ into account simultaneously all the sets of parameters compatible with the preference information 
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provided by the DM considering a necessary and a possible preference relation. 
Keywords: PROMETHEE methods, Interaction between criteria, Bipolar Choquet integral. 
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1 Introduction 

In many decision making problems (for a survey on Multiple Criteria Decision Analysis (MCDA) see [5]), 
alternatives are evaluated with respect to a set of criteria being not mutually preferentially independent 
?— i (see [22J). In fact, in most cases, the criteria present a certain form of positive (synergy) or negative 
(redundancy) interaction. For example, if one likes sport cars, maximum speed and acceleration are very 
important criteria. However, since in general speedy cars have also a good acceleration, giving a high 
weight to both criteria can over evaluate some cars. Thus, it seems reasonable to give maximum speed 
and acceleration considered together a weight smaller than the sum of the two weights assigned to these 
criteria when considered separately. In this case we have a redundancy between the criteria of maximum 
speed and acceleration. On the contrary, we have a synergy effect between maximum speed and price 
because, in general, speedy cars are also expensive and, therefore, a car which is good on both criteria is 
very appreciated. In this case, it seems reasonable to give maximum speed and price considered together 
a weight greater than the sum of the two weights assigned to these criteria when considered separately. In 
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these cases, the aggregation of the evaluations is done by using non-additive integrals the most known of 
which are the Choquet integral [3] and the Sugeno integral [20] (for a comprehensive survey on the use of 
non-additive integrals in MCDA see P IT21 IT3] ) . 

In many cases, we have also to take into account that the importance of criteria may also depend on the 
criteria which are opposed to them. For example, a bad evaluation on aesthetics reduces the importance of 
maximum speed. Thus, the weight of maximum speed should be reduced when there is a negative evaluation 
on aesthetics. In this case, we have an antagonism effect between maximum speed and aesthetics. 
Those types of interactions between criteria have been already taken into consideration in the ELECTRE 
methods [5]. In this paper, we deal with the same problem using the bipolar Choquet integral [1U|, [TT] 
applied to the PROMETHEE I and II methods [H[2]. 

This article extends the short paper published by the authors in [3] with respect to which we added the 
description of the bipolar PROMETHEE I method, the proofs of all theorems presented in [3] and a didactic 
example in which we apply the bipolar PROMETHEE methods and the Robust Ordinal Regression (ROR) 
[16j being a family of MCDA methods taking into account simultaneously all the sets of preference param- 
eters compatible with the preference information provided by the Decision Maker (DM) using a necessary 
and a possible preference relation. 

The paper is organized as follows. In the next section we recall the basic concepts of the classical PROMETHEE 
methods; in section 3 we introduce the bipolar PROMETHEE methods; the elicitation of preference infor- 
mation permitting to fix the value of the preference parameters of the model (essentially the bicapacities of 
the bipolar Choquet integral) is presented in section 4; in the fifth section we apply the ROR to the bipolar 
PROMETHEE methods; a didactic example is presented in section 6 while the last section provides some 
conclusions and lines for future research. 

2 The classical PROMETHEE methods 

Let us consider a set of actions or alternatives A = {a,b,c, . . .} evaluated with respect to a set of criteria 
G = {gi, . . .,g n }, where g 5 : A -»• R, j G J = {1, . . . , n} and \A\ = m. PROMETHEE [fl El is a well- 
known family of MCDA methods, among which the most known are PROMETHEE I and II, that aggregate 
preference information of a DM through an outranking relation. Considering for each criterion gj a weight 
Wj (representing the importance of criterion gj within the family of criteria G) , an indifference threshold qj 
(being the largest difference dj(a,b) = gj(a) — gj{b) compatible with the indifference between alternatives a 
and b), and a preference threshold pj (being the minimum difference cL-(a, b) compatible with the preference 
of a over b), PROMETHEE methods (from now on, when we shall speak of PROMETHEE methods, we 



shall refer to PROMETHEE I and II) build a non decreasing function Pj(a, b) of dj(a, b), whose formulation 
(see [1] for other formulations) can be stated as follows 



Pj{a,b) 







if dj (a, b) < qj 



dj (a,b)—qj 

1 if dj(a, b) > pj 



The greater the value of Pj(a,b), the greater the preference of a over b on criterion gj. For each ordered 
pair of alternatives (a, b) G A x A, PROMETHEE methods compute the value ir(a,b) = Yljej w jPj( a ib) 
representing how much alternative a is preferred to alternative b taking into account the whole set of criteria. 
It can assume values between and 1 and obviously the greater the value of n(a, b), the greater the preference 
of a over b. 

In order to compare an alternative a with all the other alternatives of the set A, PROMETHEE methods 
compute the negative and the positive net flow of a in the following way: 

(f>~{a) = — 3_ Y] n(b,a) and 0+(a) = — l — V vr(a,6). 
m — 1 ^-^ m — 1 

b£A\{a} beA\{a} 

These flows represent how much the alternatives of A \ {a} are preferred to a and how much a is preferred 
to the alternatives of A \ {a}. For each alternative a £ A, PROMETHEE II computes also the net flow 
(j)(a) = 4> + (a) — 4>~ (a) . On the basis of the positive and the negative flows, PROMETHEE I provides a partial 
ranking on the set of alternatives A, building a preference (V 1 ), an indifference (I 1 ) and an incomparability 
(TZ 1 ) relation. In particular: 



aV J b iff 



zZ T b iff 



*+(a) > $+(6), 
*-(a) < 

$+(a) - $~(a) > $+(6) - $-(b) 
$+(a) = $+(6), 
$-(a) = $-(6) 



a7?/6 otherwise 



On the basis instead of the net flows, the PROMETHEE II method provides a complete ranking on the 
set of alternatives A defining, in a natural way, a preference (V ) and an indifference (X ) relation for 
which aV n b iff $(a) > $(6) while al n b iff $(a) = $(6). 



3 The bipolar PROMETHEE methods 

In order to extend the classical PROMETHEE methods to the bipolar framework, we define for each criterion 
9j, j £ the bipolar preference function ; A x A — > [— 1, £ J' in the following way: 



Pf(a,b)=P j (a,b)-P j (b,a) = { 



Pj(a,b) if Pj(a,b) > 

(1) 

-Pj(b,a) if Pj(a,b) = 



It is straightforward proving that Pf(a, b) = —P^(b, a) for all j G J and for all pairs (a, b) G A x A. 

In this section we propose to aggregate the bipolar vector P B (a,b) = [Pf (a,b), . . . ,P^(a,b)] through 
the bipolar Choquet integral. 

The bipolar Choquet integral is based on a bicapacity [lOj [11], being a function £ : P{J) — > [—1, 1], where 
P(J) = {(C, D) : C, D C J and C n D = 0}, such that 

• £(0, J) = -l,p,(J,0) = 1, £(0,0) = (boundary conditions), 

• for all (C, D), (F, F) G P(J), if C C F and D D F, then £(C, D) < £(F, F) (monotonicity condition). 
According to (THUS], we consider the following expression for a bicapacity £: 

£(C, D) = fj,+(C, D) - (j,-(C, D), for all (C, D) G P(J) (2) 
where : F(jT) — > [0, 1] such that: 

fi + (J, 0) = 1, /i+(0, 5) =0,VBC J, (3) 
/i-(0, l 7) = l, ^(B,0) = O, VBCJ, (4) 



M +(C, D) < ^+(C U {j} , £>), V(C U {j} , D) G P(J), Vj G J, 
M + (C,£>) >// + (C,F>U{i}), V(C,5U{j})eP(J), Vj GJ 

M -(C, £>) < /i-(C, £) U {j}), V(C, I? U {j}) G F(J), Vj G J, 
/i-(C, £>) > M"(C U {j} , D), V(C U {j} , £>) G P(J), Vj G J 

Let us observe that © are equivalent to the constraint 

V + {C, D) < fi + (E, F), for all (C, D), (F, F) G P(J) such that C C F and F D F, 



(5) 



(6) 



while © are equivalent to the constraint 



H~(C, D) < n~(E, F), for all (C, D), (E, F) G such that C D £ and D Q F. 

The interpretation of the functions /x + and ^~ is the following. Given the pair (a, 6) € j4 x A, let us 
consider (C, L>) € P{J) where C is the set of criteria expressing a preference of a over 6 and -D the set of 
criteria expressing a preference of b over a. In this situation, n + (C, D) represents the importance of criteria 
from C when criteria from D are opposing them, and fi~(C,D) represents the importance of criteria from 
D opposing C. Consequently, fi(C, D) represents the balance of the importance of C supporting a and D 
supporting b. 

Given (a, b) G A x A, the bipolar Choquet integral of P B (a,b) with respect to the bicapacity ft can be 
written as follows 

Ch B (P B (a,b),ft) = I fx({j € J : Pf(a,b) > t},{j G J : Pf(a,b) < -t})dt, 
Jo 

while the bipolar comprehensive positive preference of a over b and the comprehensive negative preference 
of a over b with respect to the bicapacity p, are respectively: 

Ch B+ (P B (a,b),fi) = /V({j S J : Pf(a,b) >t},{jej: P B (a,b) < -t})dt, 
Jo 

Ch B -(P B (a,b),fi) = G J : Pf(a,b) >t},{j£j: P B (a,b) < -t})dt, 

Jo 

where [i + and have been defined before. 

Prom an operational point of view, the bipolar aggregation of P B (a,b) can be computed as follows: for 
all the criteria j G J, the absolute values of these preferences should be re-ordered in a non-decreasing way, 
as follows: \P { B } (a,b)\ < \P { %(a,b)\ < . . . < \P^{a, b)\ < . . . < \P^ n) (a,b)\. 

The bipolar Choquet integral of P B (a,b) with respect to the bicapacity ft, can now be determined: 



Ch B (P B (a,b),ft) = J] \P^(a,b)\h(C (j) (a,b),D (j) (a,b)) - ft {C u+1) (a,b), D u+1) (a,b))] (7) 



where P B (a,b) = [P B (a,b), j G j\, J> = {j G J : \P§ } (a,b)\ > 0}, C {j) (a,b) = {i G J> : P B (a,b) > 
IP^M)!}, D {j) (a,b) = {i G J> : -P B (a,b) > \P B } (a,b)\} and C {n+1) (a,b) = D (n+1) (a,b) = 0. 
We give also the following definitions: 



Ch B+ {P"{a,b)^ + ) 



j€J> 



P ( %(a,b)\\v + (C u) (a,b),D U) (a,b)) - fi + (C u+1) (a, b), D {j+l) (a, b)) } , (8) 



Ch B -(P B (a,b),^) = Y, \P { B 3) (a,b)\[^ (C U) (a,b),D {j) (a,b))-v- (C (j+1) (a, b), D {j+1) (a, b)) ] . (9) 

ChP \P B (a,b), /t) gives the comprehensive preference of a over b and it is equivalent to n(a, b) — n(b,a) = 
P c (a,b) in the classical PROMETHEE method while Ch B+ (P B (a,b), and Ch B - (P B (a,b), p~) give, 
respectively, how much a outranks b (considering the reasons in favor of a) and how much a is outranked 
by b (considering the reasons against a). 

From the definitions above, it is straightforward proving that, for all a, b £ A, 

Ch B (P B (a,b),fi) = Ch B+ (P B {a,b),n + )-Ch B -(P B ( y a,b),fi-) (10) 

Using equations (J7]), (|8|) and Q, we can define for each alternative a 6 A the bipolar positive flow, the 
bipolar negative flow and the bipolar net flow as follows: 

cp B+ (a) = ^— V Ch B +(P B (a,b),»+) (11) 
m — 1 

beA\{a} 

0*- (a)= 1 Ch B -(P B (a,b),^) (12) 

m — 1 

beA\{a} 



^ B (a) 



l — Y, Ch B (P B (a,b),fi) (13) 



m 

b£A\{a} 



By equation (fTU|) . it follows that (j) B (a) = 4> B+ (a) — 4> B ~(a) for each a € A 

Analogously to the classical PROMETHEE I and II methods, using the positive, the negative and the 
net bipolar flows we propose the bipolar PROMETHEE I and the bipolar PROMETHEE II methods. Given 
a pair of alternatives (a, b) € A x A, the bipolar PROMETHEE I method defines a partial order on the set 
of alternatives A considering a preference (Vg), an indifference (Xg) and an incompar ability (R<b) relation 
defined as follows: 



aV B b iff 



al B b iff 



a1Z B b otherwise 



$ B -(a) < 

$ B +(a) - $ B ~(a) > <S> B+ (b) - $ B ~(&) 
^ $ B -(a) = $ B -(6) 



Given a pair of alternatives (a, b) £ A x A, the bipolar PROMETHEE II method provides, instead, a 
complete order on the set of alternatives A, defining the a preference (V B ) and an indifference {I B ) 
relations as follows: aPgb iff <S> B (a) > $(6), while algb iff <S> B (a) = <b B (b). 

3.1 Symmetry conditions 

Because Ch B (P B (a, 6), /t) is equivalent to 7r(a, &)— 7r(6, a) = P c (a, 6) in the classical PROMETHEE method, 
it is reasonable expecting that, for all a,b € A, ChP (P B (a,b), ft) = —Ch B (P B (b,a),fi). The following 
Proposition gives conditions to satisfy such a requirement: 

Proposition 3.1. Ch B (P B (a, &),£) = -Ch B (P B (b, a), /}) for all possible a,b, iff fi(C,D) = 
-fi{D,C) for each (C,D)eP(J). 



Proof. Let us prove that if /t(C, D) 



-fi(D,C) for each (C,D) € P(<7), then Ch B {P B (a,b), ft) 



-Ch B (P B (b,a),fi). As noticed, P B (a,b) = -P B (b,a) for all j G J, and consequently \P^(a,b)\ 
\-P B } (b,a)\ = \P^(b,a)\ for all j € X 
From this, it follows that: 



(a) C 0) (a,6) = {iej> : if (a, 6) > \P^(a,b)\} = {i £ J> : -if (6, a) > \P^(b,a)\} 
= D(j)(b,a); 

(/?) %(a,6) = { l Gj > : -P B (a,b)>\P B ) (a,b)\} = {iGj> : if (6, a) > |P ( f } (6, a)|} 
= C(j) a). 

From (a) and (/3) we have that 

( 7 ) <^ B (P S (a, &),£) = 
j"e.7> 

= Yl l P (?)(Ml[M^(;o(M, c (;0(M) -APo+i)(M),C 0+1) (&,a)) 
j'e.7> 



Since pt{C,D) = -fi(D,C), V(C, D) € P(J), from (7) we have that, 

(8) Ch B (P B (b,a),ft) = 

= \ P S)( b ' a )\ KC(j){b,a),D {j) (b,a)) - fi{C {j+1) {b,a),D {j+1) (b,a)) 

= Y l p (f)( 6 ' a )l - A(^0)(^a),C (j )(6,a)) + fi{D {j+1) (b,a),C {j+ i } (b,a)) 
jej> 

= -Ch B (P B (a,b),fi). 



Let us now prove that if Ch B (P B (a,b), ft) = -Ch B (P B (b,a), ft), then £l(C,D) 
consider the pair (a, b) such that, 



-fi(D, C). Let us 



Pf(a,b) = t 



1 if jGC 
-1 if j GD 
otherwise 



(14) 



In this case we have that Ch B (P B (a,b), ft) = jl{C,D) and Ch B (P B (b,a), ft) = fi(D,C). Thus if 
Ch B (P B (a,b),fi) = -Ch B (P B (b,a),fi), by (iv) we obtain that /x(C,D) = -fi(D,C) and the proof is 
concluded. □ 

Analogously, because Ch B+ (P B (a, b),/j, + ) represents how much a outranks b and Ch B ~ (P B (b, a),fi~) repre- 
sents how much b is outranked by a, it is reasonable expecting that Ch B+ (P B (a, b), /i + )=Ch B ~ (P B (b, a), 
Sufficient and necessary conditions to get this equality are given by the following Proposition. 

Proposition 3.2. Ch B+ (P B (a, b), n+) = Ch B -(P B (b, a), n~) for all possible a,b, iff n + (C,D) = 
fJ,-(D,C) for each (C,D)£P(J). 

Proof. Analogous to Proposition 13.11 □ 



Reminding equation (jlOH . the Corollary follows. 

Corollary 3.1. Ch B {P B {a,b) : ft) = -Ch B (P B (b, a), ft) for all possible a,b, if fi + (C,D) = fi~(D,C) 
for each (C,D)eP(J). 

Proof. This can be seen as a Corollary both of Proposition 13.11 and Proposition 13.21 In fact, 

• n + (C,D) = fjL'(D,C) for each (C,D) € P{J) implies that ft{C,D) = -fi{D,C) for each (C,D) € 
P{J), and by Proposition 13.11 it follows the thesis. 

• fi + (C,D) = n~(D,C) for each (C,D) € P{J) implies that Ch B+ (P B (a, b), //+) = Ch B -(P B (b,a), fj,~) 
(by Proposition 13. 2p and from this it follows obviously the thesis by equation (|T0]) . 



□ 



3.2 The 2-additive decomposable bipolar PROMETHEE methods 

As seen in the previous section, the use of the bipolar Choquet integral is based on a bicapacity which 
assigns numerical values to each element P{J"). Let us remark that the number of elements of P{J) is 3 n . 
This means that the definition of a bicapacity requires a rather huge and unpractical number of parameters. 
Moreover, the interpretation of these parameters is not always simple for the DM. Therefore, the use of the 
bipolar Choquet integral in real-world decision-making problems requires some methodology to assist the 
DM in assessing the preference parameters (bicapacities) . Several studies dealing with the determination of 
the relative importance of criteria were proposed in MCDA (see e.g. |19|). The question of the interaction 
between criteria was also studied in the context of MAUT methods |17j . 

In the following we consider only the 2-additive bicapacities [10], [7], being a particular class of bicapacities. 

3.3 Defining a manageable and meaningful bicapacity measure 

According to [HI, we §i ye the following decomposition of the functions fi + and yT previously defined: 
Definition 3.1. 

. ^+(C 5J D) = J>+({j},0)+ £ a+({j,k}J) + Y, « + (0'M*» 
jec {j,k}QC jec. keD 

. A r(C,Z>) = 5>-(0,{i})+ £ a-(0, {;,*})+ Yl «"({#>{*» 
jeD 0',fc}cD jec, k&D 

The interpretation of each a ± (-) is the following: 

• a + ({j},0), represents the power of criterion gj by itself; this value is always non negative; 

• a + ({j, k}, 0), represents the interaction between gj and g^, when they are in favor of the preference 
of a over 6; when its value is zero there is no interaction; on the contrary, when the value is positive 
there is a synergy effect when putting together gj and ; a negative value means that the two criteria 
are redundant; 

• a + ({j}, {k}), represents the power of criterion g^ against criterion gj, when criterion gj is in favor of 
a over b and g^ is against to the preference of a over b; this leads always to a reduction or no effect on 
the value of /i + since this value is always non-positive. 

An analogous interpretation can be applied to the values a~(0, {j}), a~(0, {j, k}), and a~({j}, {k}). 

In what follows, for the sake of simplicity, we will use at, a+ fc , adj fc instead of a + ({j}, 0), a + ({j, k}, 0) and 
a + ({j}, {k}), respectively and aj , aj k , aj^ k instead of a~(0, {j}), a~(0, {j, k}) and a~({j}, {k}), respectively. 
In this way, the bicapacity /t, decomposed using fi + and yT of Definition 13.11 has the following expression: 



£(C,Z>) = im+(C,D)-iT(C,D) 



E4-E°7+ E E «7* + E 4*- E 

jec j&D {j,fc}cc {;,fc}co jec, keD jec, keD 



a j\k 



We call such a bicapacity fi, a 2-additive decomposable bicapacity . (An analogous decomposition has been 
proposed directly for ft without considering /i + and [i~ in [8]). 

Considering these decompositions for the functions /i + and fi~ , the monotonicity conditions ([!]), ([6]) and the 
boundary conditions (El), (Hj) have to be expressed in function of the parameters aj" , at, at, , a7, a~i,, a^ 
as follows: 

Monotonicity conditions 

1) ^(C,D)<^(CU{j},D), VjSj, V(CU{j},D)GP(J) 



° 4 + E a tk + E a t\k ^ °« V i e J, V(C U {i}, £>) G P( J) 
fceC " fceD 

2) M +(C, D) > /x+(C, D U {j}), Vjej, V(C, D U {j}) G P(J) 

^ 4\j < °> V i G J, V(C, D U {j}) G PG7) 

fcGC 

being already satisfied because a^. < 0, VA, j G J ,k ^ j. 

3) f i-(C,D)< fJ> -(C,DU{j}), Vj€j,V(C,DU{j})eP(J) 

a J + E °7* + E ^ °> v J G ^ v ( c < 15 u 0'}) e W 

4) //"(CD) >/i-(CU{j},D), VjGj, V(CU{j},D)GP(J) 

° E a 7fc ^ °> v i G ^' v ( c u ft >> D ) G p ( J ) 

feeD 

being already satisfied because a^j fc < 0, Vj, A; G J , j ^ A;. 



Conditions 1), 2), 3) and 4) ensure the monotonicity of the bi-capacity, /}, on J, obtained as the difference 
of /i + and that is, 



V (C,D), (E,F) G P(J) such that C^E,DQF, £l(C, D) > fi(E, F). 
Boundary conditions 

1. M + G7,0) = l,i-e., J>t + E a J = 1 

2. M -(0, J) = 1, i.e., J] a" + Yl a Jk = 1 

jejr {j,k}Qj 



3.4 The 2-additive bipolar Choquet integral 

The following theorem gives an expression of Ch B+ (x, and Ch B ~(x, considering a 2-additive de- 
composable bicapacity /i. 



Theorem 3.1. Given a 2-additive decomposable bicapacity p,, then for all x £ 



1. Ch B+ (x,/j, + ) = ^ ajxj + ^ a+ k mm{xj,x k } + ^ a1f fc mm{xj, -x k } 

j€j,Xj>0 j,k£j ,j^k,Xj,Xk>0 j,keJ,j^k,Xj>0,x k <0 

2. Ch B ~(x,fi~) = - ^ a j x j~ E aj k m.ax{xj,x k } - ^ aTj fe max{-Xj, x k } 

jej,xj<0 j,kej,j^k,xj,x k <0 j,k£j,j^k,Xj>0,x k <0 

Proof. We shall prove only part 1. Proof of part 2. can be obtained analogously. 
If the bicapacity ft is 2— additive decomposable, then 



Ch B+ (x,fi + ) = Y \ x (j)\[^ + ( C {3)^ D (j)) ~ V + ( C (j+i), D (j+i)) 



Let us remark that, 



= E hi) I 

jej> 



E 4- E 0+ 

kej> ,xk>\x(j)\ kej> ,xfc>|xy + i)| 

+( E <k- E a ^)+ 

h,k€j> ,hj=k,x h ,x k >\x(j)\ h,k€j> ,fc^fc,x;,,Xfc>|xy+i)| 



+ 



E 



E a h\k) 



h,kej> ,h^k,x h ,-x k >\x(j } \ h,keJ>,h^k,x h ,-x k >\x (j+1 - ) \ 



u ' ' E a * E "/ ) 

fce^jXfc^Ixy)! fc€>7 > ,x fc >|x(., +1 )| 



E 



a k if < |xri 



fe€j r> ,a; fc = |a: (i) | 



otherwise 



V) 



E 



E 



% i = < 



k£j> ,-Xfc>|xy)| kej> ,-x k >\X(j + i)\ 



E a k if l x (i)l < 

kej> ,-x k = \x^)\ 

otherwise 



c) 



E a hk ~ E "l'k 

h,kej>,h^k, h,k(£j > ,h^k, 
X h ,X k >\x(j)\ x h ,x k >\x (j + 1) 



E a hk if \ x u)\ < \ x u- 



h,k&J>,h^k, 
m'm{x h ,x k }=\x^\ 



otherwise 



Considering a) — c) we get that: 



X) 



E 



jej>, 

\ x u)\<\ x u+i)\ 



E 4 

fce l 7 > ,x fc =|a:( j) 



E ■ 

h,k£j>,h^k, 
min{x h ,x k }=\x u) \ 



lik 



E a h|* 
h,kej>,h^k, 
min{x h ,-x k }=\x^\ 



and from this it follows the thesis. 



□ 



In the following, we provide the symmetry conditions of Propositions 13,11 and 13,21 in function of the 
parameters af,aj, a+ , a~ k , a^ k and a~ fc . 

Proposition 3.3. Given a 2-additive decomposable bicapacity fi, then £t(C, D) = —fi(D, C) for each (C, D) G 
P(J) iff 

1. for each j e J, Oj = aj , 

2. for each {j, k} C J, a+ k = aj k , 

3. for each j,kej,j^k, a+ k - a~ k = a", . - a+ r 
Proof. First, let us prove that 

(a) £(C,£>) = -KD,C) 

implies 1., 2. and 3. For each j € J", 

(6) A({i},0) = 4 and A(0,O'}) = -aj 
From (a) and (6) we have, 



4 = A({i},0) = -A(0,{i})=«7 



which is 1. 

For each {j, k} C J we have that, 

(c) £({j, k}, 0) = a+ + a£ + a+ fe and /i(0, {j, k}) = -aj - aj; - aj k 

Being p,({j, k},$) = — /i(0, {j, k}), and being at = a~ and = by 1., we have that for each 
{j,k} C J" , a+ fc = a~ fc , i.e. 2. 

For all j,k G with j 7^ fe, we have: 

£(0'}i W) = 0+ - ^ + aj fc - aj ]k 
£({*}. {j}) = a+ - aj + a+j - a % . 
Being fi({j},{k}) = — /}({&}, {j}) and having proved that aj = aJ,Vj, we obtain that at fc — a^ k 



-°fcii + *ii u - 3 - 



It is straightforward to prove that 1., 2., and 3. imply fi(C,D) = —fi(D,C). 

□ 

Corollary 3.2. Given a 2-additive decomposable bicapacity fx, Ch B (P B (a,b), /t) = —Ch B (P B (b,a),jl) for 
all a,b £ A iff 

1. for each j G J , alj = a~, 

2. /or eac/i {j, A;} C J, a+ fc = a~ fc; 

5. for each j,kej,j^k, a+ fe - a~ fe = a^. - aj.. 
Proof. It follows by Propositions 13.31 and 13.11 □ 
Proposition 3.4. Given a 2-additive decomposable bicapacity fi, then fi + (C,D) = fi~(D,C) for each 

(C,D)eP(j) iff 

1. for each j G J, a'j = aj , 

2. for each {j, k} C J, a+ fc = aj k , 

3. for each j, k G J, j ^ k, a^ k = a~^. 

Proof. Analogous to Proposition 13.31 □ 

Corollary 3.3. Given a 2-additive decomposable bicapacity fi, Ch B+ (P B (a,b), /U + ) = Ch B ~ (P B (b, a), /U~) 
for all a,b G A iff 



1. for each j € J , a,j 



+ 



= a, a 



'j ' 



2. for each {j, k} C J , = a 



3. for each j, k G J, j ^ k, aj, k = a 



'k\j- 



Proof. It follows by Propositions 13.41 and 13.21 



□ 



Because the first two conditions of Proposition 13.11 are the same of the first two conditions of Proposition 
13.21 but the third condition of Proposition 13.21 implies the third one of Proposition 13.11 in order to get both 



Ch B (P B {a,b),fi) = -Ch B {P B (b,a),fi) and Ch B+ (P B {a, b), fi+) = Ch B ~{P B (b,a), n~) for all a,b e A, we 



impose that shoul be fulfilled the conditions in Proposition 13.21 

4 Assessing the preference information 

On the basis of the considered 2-additive decomposable bicapacity (j,, and holding the symmetry condition 
in Corollary 13.31 we propose the following methodology which simplifies the assessment of the preference 
information. 

We consider the following information provided by the DM and their representation in terms of linear 
constraints: 

1. Comparing pairs of actions locally or globally. The constraints represent some pairwise comparisons on 
a set of training actions. Given two actions a and b, the DM may prefer a to b, b to a or be indifferent 
to both: 

(a) the linear constraint associated with aVb (a is locally preferred to b) is: 



(b) the linear constraints associated with aV l b (a is preferred to b with respect to the bipolar 



Ch B (P B {a,b),fi) > 0; 



PROMETHEE I method) are: 



$ B+ (a) > 
$ B -(a) < $ B -{b), 

$ B +(a) - $ B -(a) > $ B +(b) - $ B -(b) 



> 



(c) the linear constraint associated with aP b (a is preferred to b with respect to the bipolar 
PROMETHEE II method) is: 

$ B (a) > § B {b) 

(d) the linear constraint associated with alb (a is locally indifferent to b) is: 

Ch B (P B (a,b),fi) = 

(e) the linear constraints associated with aL l b (a is indifferent to b with respect to the bipolar 
PROMETHEE I method) are: 

$ s +(a) = $ B +(6), 1 
$ B -(a) = <S> B -(b), J 

(f) the linear constraint associated with al n b (a is indifferent to b with respect to the bipolar 
PROMETHEE II method) is: 

$ B (a) = $ B {b) 

Comparison of the intensity of preferences between pairs of actions. The constraints represent some 
pairwise comparisons between pairs of alternatives on a set of training actions. Given four actions a, 
b, c and d: 

(a) the linear constraints associated with (a, b)V(c, d) (the local preference of a over b is larger than 
the local preference of c over d) is: 

Ch B (P B (a,b),jl) > Ch B (P B (c,d),fi) 

(b) the linear constraints associated with (a, b)l(c, d) (the local preference of a over b is the same of 
local preference of c over d) is: 

Ch B (P B (a,b),fi) = Ch B (P B (c, d),jl) 

Importance of criteria. A partial ranking over the set of criteria J may be provided by the DM: 

(a) criterion gj is more important than criterion g^, which leads to the constraint aj > a^; 

(b) criterion gj is equally important to criterion which leads to the constraint aj = a^. 



4. The sign of interactions. The DM may be able, for certain cases, to provide the sign of some inter- 
actions. For example, if there is a synergy effect when criterion gj interacts with criterion gk, the 
following constraint should be added to the model: ajk > 0. 

5. Interaction between pairs of criteria. The DM can provide some information about interaction between 
criteria: 

a) if the DM feels that interaction between gj and g^ is greater than the interaction between g p and 
g q , the constraint should be defined as follows: \ajk\ > \a pq \ where in particular: 

• if both couples of criteria are synergic then: ajk > a pq , 

• if both couples of criteria are redundant then: ajk < a pq , 

• if (j, k) is a couple of synergic criteria and (p, q) is a couple of redundant criteria, then: 

• if (j, k) is a couple of redundant criteria and (p, q) is a couple of synergic criteria, then: 

fljfc > a pq . 

b) if the DM feels that the strength of the interaction between gj and g^ is the same of the strength 
of the interaction between g p and g q , the constraint will be the following: \ajk\ = \a pq \ and in 
particular: 

• if both couples of criteria are synergic or redundant then: ajk = a pq , 

• if one couple of criteria is synergic and the other is redundant then: ajk = —a pq , 

6. The power of the opposing criteria. Concerning the power of the opposing criteria several situations 
may occur. For example: 

a) when the opposing power of gk is larger than the opposing power of gh, with respect to gj, 
which expresses a positive preference, we can define the following constraint: a1j fc < a1j ft (because 

< and aj^ h < for all j, k with j ^ k); 

b) if the opposing power of gk, expressing negative preferences, is larger with gj rather than with 
gh, the constraint will be a^j fc < a^ fc . 

4.1 A linear programming model 

All the constraints presented in the previous section along with the symmetry, boundary and monotonicity 
conditions can now be put together and form a system of linear constraints. Strict inequalities can be 
converted into weak inequalities by adding a variable e. It is well-know that such a system has a feasible 



solution if and only if when maximizing e, its value is strictly positive [T7j. Considering constraints given 
by Corollary 13.31 for the symmetry condition, the linear programming model can be stated as follows (where 
jVk means that criterion gj is more important than criterion g^; the remaining relations have a similar 
interpretation): 



Max e 

Ch B (P B (a,b),fa)>e if aVb, Ch B (P B (a,b), fa) = if alb, 

<S> B +(a) > <S> B+ (b), ) , 

r ■Z B +(a) = <t> B +(b), I 

$ fl -(a) < $ B -(b), } if aV' B b } if aX' B b 

* B -(o) =$ s -(6) 

<I> s +(a) -$ s ~(a) > $ s +(fe) - $ s -(6) + e J ' 
<I> s (a) > $ s (o) + e if aPj/b <I>- B (a) = if aX^fe 

Ch B (P B (a,b),fa)>Ch B (P B (c,d),fa) + e if (a,b)V(c,d), Ch B (P B (a,b), fa) = Ch B (P B (c,d), fa) if (a,b)X(c,d), 

a j ~ a k > £ if ,?"Pfc> a_, = aft if jXfc, 

l a jfel ~~ a p<? > e if {j, fc}'P{p, g}, (see point 5. a) of the previous subsection ) 

\ a ik\ = \ a pq\ if {ii k}I{p, q}, (see point 5.b) of the previous subsection ) 

a jk > e if there is synergy between criteria j and k, 

djk < — s if there is redundancy between criteria j and k, 

a jk = if criteria j and k are not interacting, 

Power of the opposing criteria of the type [6] 

af,, — a\ > e, a~,, — a~, > e. 

j\k j\p — j\k j\p — ' 

of,, — a~^, > e, a~,, —a~,, > e, 

j\k p\k — 7 j\k p\k — ' 

Symmetry conditions (Proposition 13. 3"t : 
a + ,, = a~, . , V j.k 6 J,j ^ k 

j\k k\j' ,J ' 

Boundary and monotonicity conditions: 

jej {i,fe}Cj- 

^ > V j G J, a+ k , a~ k <0 Vj,keJ, 



keC k£D 

+ E a h\ 3 > °> V j e V(C, £> u {j}) e P(j). 



*3 ' 

keD hec 



4.2 Restoring PROMETHEE 

The condition which allows to restore the classical PROMETHEE methods is the following: 

1. Vj, k € J, a jk = a^ k = aj\ k = 0. 

If Condition 1. is not satisfied and the following condition holds 

2. Vj,keJ,a+ k = aJ lk = 0, 



then the comprehensive preference of a over b is calculated as the difference between the Choquet integral 
of the positive preferences and the Choquet integral of the negative preferences, with a common capacity fi 
on J for the positive and the negative preferences, i.e. there exists fi : 2^ — > [0, 1], with ^(0) = 0, n{J) = 1, 
and fi(A) < fi(B) for all A C B C J, such that 

Ch B (P B {a,b),p,) = [ G J : Pf(a,b) > t})dt - [ G J : Pf(a,b) < -t})dt. 

Jo Jo 

We shall call this type of aggregation of preferences, the symmetric Choquet integral PROMETHEE method. 
If neither 1. nor 2. are satisfied, but the following condition holds 

3. Vj,feej,a+ fc = a- |3 , 

then we have the Bipolar PROMETHEE methods. 



4.3 A constructive learning preference information elicitation process 

The previous Conditions 1.-3. suggest a proper way to deal with the linear programming model in order to 
assess the interactive bipolar criteria coefficients. Indeed, it is very wise trying before to elicit weights con- 
cordant with the classical PROMETHEE method. If this is not possible, one can consider a PROMETHEE 
method which aggregates positive and negative preferences using the Choquet integral. If this is not possi- 
ble, one can consider the bipolar symmetric PROMETHEE method. If, by proceeding in this way, we are 
not able to represent the DM's preferences, then we can take into account a more sophisticated aggregation 
procedure by using the bipolar PROMETHEE method. This way to progress from the simplest to the most 
sophisticated model can be outlined in a four steps procedure as follows: 

1. Solve the linear programming problem 



Max e = £\ 

E A» ) (15) 

r El 

a 3k = tf\ k = %\ k = 0, Vj,kej J 

adding to E aR the constraint related to the previous Condition 1. If E\ is feasible and E\ > 0, then the 
obtained preferential parameters are concordant with the classical PROMETHEE method. Otherwise, 

2. Solve the linear programming problem 



Max e = e 2 

E A* ) (16) 

>E 2 

a l\k = a j\k = °> V ^ £ ^ J 

adding to the constraint related to the previous Condition 2. If E 2 is feasible and e 2 > 0, then 
the information is concordant with the symmetric Choquet integral PROMETHEE method having a 
unique capacity for the negative and the positive part. Otherwise, 

3. Solve the linear programming problem 

Max e = £3 

(17) 

E A * 

If E3 is feasible and £3 > 0, then the information is concordant with the bipolar PROMETHEE 
method. Otherwise, 

4. We can try to help the DM by providing some information about inconsistent judgments, when it is 
the case, by using a similar constructive learning procedure proposed in [IB]. In fact, in the linear 
programming model some of the constraints cannot be relaxed, that is, the basic properties of the 
model (symmetry, boundary and monotonicity conditions). The remaining constraints can lead to an 
infeasible linear system which means that the DM provided inconsistent information about her/his 
preferences. The methods proposed in |18j can then be used in this context, providing to the DM 
some useful information about inconsistent judgments. 

5 ROR and Bipolar PROMETHEE methods 

In the above sections we dealt with the problem of finding a bicapacity restoring preference information 
provided by the DM in case where multiple criteria evaluations are aggregated by Bipolar PROMETHEE 
method. Generally, there could exist more than one model (in our case the model will be a bicapacity, but 
in other contexts it could be a utility function or an outranking relation) compatible with the preference 
information provided by the DM on the training set of alternatives. Each compatible model restores the 
preference information provided by the DM but two different compatible models could compare the other 
alternatives not provided as examples by the DM in a different way. For this reason, the choice of one 
of these models among those compatible could be considered arbitrary. In order to take into account not 



only one but the whole set of models compatible with the preference information provided by the DM, 
we consider the ROR [16]. This approach considers the whole set of models compatible with preference 
information provided by the DM building two preference relations: the weak necessary preference relation, 
for which alternative a is necessarily weakly preferred to alternative b (and we write a ^ N b), if a is at least 
as good as b for all compatible models, and the weak possible preference relation, for which alternative a is 
possibly weakly preferred to alternative b (and we write a ^ p b), if a is at least as good as b for at least one 
compatible model. 

Considering the bipolar flows (jlip - (|13p and the comprehensive Choquet integral in equation (jlOp . given the 
alternatives a,b € A, we say that a outranks b (or a is at least as good as b): 

• locally, if Ch B {P B (a,b),fi) > 0; 

• globally and considering the bipolar PROMETHEE I method, if $ B +(a) > <S> B+ (b), <S> B -(a) < <S> B -(b); 

• globally and considering the bipolar PROMETHEE II method, if $ B {a) > $ B (b). 

To check if a is necessarily preferred to b, we look if it is possible that a does not outrank b. Locally, this 
means that it is possible that there exists a bicapacity fi such that Ch B (P B (a, b), fi) < 0; globally, considering 
the bipolar PROMETHEE I this means that <& B+ (a) < <$> B+ (b) or $ B_ (a) > <$> B ~(b), while considering the 
bipolar PROMETHEE II this means that $ B (a) < & B (b). Given the following set of constraints, 



}E N (a,b) 



E AR 

if one verifies the truth of global outranking: 

if exploited in the way of the bipolar PROMETHEE II method, then: 

$ B (a) +e < $ B (b) 
if exploited in the way of the bipolar PROMETHEE I method, then: 
$ B+ (a) + e<f B+ (/)) + 2Mi and <S> B - (a) + 2M 2 > $> B ~ (b) + e 
where M, E {0, 1}, i = 1, 2, and Y?i=i M i < 1 
if one verifies the truth of local outranking: 
Ch B (P B {a,b),jl) + e < 

we say that a is weakly necessarily preferred to b if E N (a,b) is infeasible or e* < where e* = maxe s.t. 
E N (a,b). 

To check if a is possibly preferred to b, we check if it is possible that a outrank b for at least one bicapacity 
fi. Locally, this means that there exists a bicapacity fi such that Ch B \P B (a,b), fl) > 0; globally, considering 



PROMETHEE I this means that $ B+ (a) > $ B+ (b) and ^ B ~(a) < while considering PROMETHEE 

II this means that <3? s (a) > Q B (b). Given the following set of constraints, 



► E p (a,b) 



E AR 

if one verifies the truth of global outranking: 

if exploited in the way of the bipolar PROMETHEE II method, then: 
$ B (a) > 

if exploited in the way of the bipolar PROMETHEE I method, then: 
$ B+ (a) > $ B+ (b) and <£> B - (a) < $ B ~(&) 
if one verifies the truth of local outranking: 
Ch B (P B {a,b),fl) > 

we say that a is weakly possibly preferred to b if E p (a,b) is feasible and e* > where e* = maxe s.t. 



E p {a,b). 



6 Didactic Example 

Inspired by an example in literature [9], let us consider the problem of evaluating High School students 
according to their grades in Mathematics, Physics and Literature. In the following we suppose that the 
Director is the DM, while we will cover the role of analyst helping and supporting the DM in (her)his 
evaluations. 

The Director thinks that scientific subjects (Mathematics and Physics) are more important than Literature. 
However, when students a and b are compared, if a is better than b both at Mathematics and Physics but a 
is much worse than b at Literature, then the Director has some doubts about the comprehensive preference 
of a over b. 

Mathematics and Physics are in some sense redundant with respect to the comparison of students, since 
usually students which are good at Mathematics are also good at Physics. As a consequence, if a is better 
than b at Mathematics, the comprehensive preference of the student a over the student b is stronger if a is 
better than b at Literature rather than if a is better than b at Physics. 

Let us consider the students whose grades (belonging to the range [0,20]) are represented in Table [Q and 
the following formulation of the preference of a over b with respect to each criterion gj, for all j = (M) 
Mathematics, (Ph) Physics, (L) Literature. 



Students 


Mathematics 


Physics 


Literature 




id 


id 


ID 


$2 








6 3 


1 Q 


J. O 


1 4 


S4 


18 


16 


15 


S<5 


15 


16 


17 


s& 


13 


13 


19 


S7 


17 


19 


15 


S8 


15 


17 


16 



Table 1: Evaluations of the students 



Pj(a,b) = < 



if gj (b)> gj {a) 
{g 3 {a)-9 3 {b))/A if 0< gj (a)- gj (b)<4 

1 otherwise 



From the values of the partial preferences Pj{a,b), we obtain the positive and the negative partial 
preferences P B (a,b) with respect to each criterion c/j, for j = M,Ph,L using the definition ([I]). Thus, to 
each pair of students (sj, Sj) is associated a vector of three elements: 

P B (si,Sj) = [Pjrf(si, Sj), Pp h (si, Sj), P^ (si, Sj)]; for example, to the pair of students (si,S2) is associated 
the vector P B (si,s 2 ) = [0.25,0.75,-0.5]. 

Let us suppose that the Dean provides the following information regarding some pairs of students: 

• student si is preferred to student S2 more than student S3 is preferred to student S4, 

• student S7 is preferred to student sg more than student S5 is preferred to student sq. 
As explained in section HI these two information are translated by the constraints: 

Ch B (P B ( Sl ,s 2 ),fi) > Ch B (P B (s 3 ,s A ),ft), and Ch B (P B (s 7 , s 8 ), /}) > Ch B {P B {s 5 , s 6 ), p) 

Following the procedure described in section 14.31 at first we check if the classical PROMETHEE method 
and the symmetric Choquet integral PROMETHEE method are able to restore the preference information 
provided by the Dean; solving the optimization problems 1151 and 1161 we get £i < and £2 < and therefore 
neither the classical PROMETHEE method nor the symmetric Choquet integral PROMETHEE method 
are able to explain the preference information provided by the Dean. Solving the optimization problem [T71 
we get this time £3 > 0; this means that the information provided by the Dean can be explained by the 
Bipolar PROMETHEE method. 

In order to better understand the problem at hand, we suggested to the Dean to use the ROR applied to 



the bipolar PROMETHEE method as discussed in the previous section. Using the first piece of preference 
information, we get the necessary and possible preference relations shown in Table [2] at local level and 



considering PROMETHEE II and PROMETHEE I. In Table 2(a), the value 1 in position means that 



S{ is necessarily locally preferred to Sj while the viceversa corresponds to the value. Analogous meaning 



have the values 1 and in in Tables 2(b) and 2(c) respectively. 

Table 2: Necessary preference relations after the first piece of preference information 
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Local 






(b) PROMETHEE II 
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II 


1 


1 


S8 





1) 








S8 


II 


















Table 3: Possible preference relations after the first piece of preference information 



(a) Local (b) PROMETHEE II (c) PROMETHEE I 
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S3 S 4 
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S3 


s 4 


S5 


so 


st 


ss 


Sl 





1 


1 


1 


1 


1 


si 


1) 


1 


1 


1 


1 


1 





1 


Sl 


1) 


1 


1 


1 


1 


1 





1 


S2 


I) 











1 





S2 


1 





1 


1 





1 





1 


S2 











1 





1 








S3 


1 


1 


1 


1 


1 1 


1 


S3 


1 


1 





1 


1 


1 


1 


1 


S3 


1 


1 





1 


1 


1 


1 


1 


S-l 


1 


1 





1 


1 1 


1 


s 4 


1 


1 








1 


1 


1 


1 


s 4 


1 


1 








1 


1 


1 


1 


S5 


1 


1 


1 1 





1 





S5 


1 


1 


1 


1 





1 





1 


S5 


1 


1 


1 


1 





1 





1 


so 


II 


1 


1 


1) 








so 


1 


1 


1 


1 











1 


so 


1 


1 


1 


1 


II 











S7 


1 


1 


1 1 


1 


1 


i 


ST 


1 


1 


1 


1 


1 


1 





1 


ST 


1 


1 


1 


1 


1 


1 





1 


ss 


1 


1 


1 1 


1 


1 





S8 


1 


1 


1 


1 


1 


1 








S8 


1 


1 


1 


1 


1 


1 









Looking at Tables [21 we underline that sj, S3 and S5 are surely the best among the eight students considered. 
In fact, S7 is necessarily preferred to five out of the other seven students both locally and considering the 
bipolar PROMETHEE II method and, at the same time, (s)he is the only student being necessarily preferred 
to some other student using the bipolar PROMETHEE I method. S3 is necessarily preferred to four out of the 
other seven students locally, and (s)he is necessarily preferred to S4 considering the bipolar PROMETHEE II 
method. At the same time, (s)he is locally possibly preferred to S7 (see Table [3]). S5 is necessarily preferred 
to S2 and sq considering the bipolar PROMETHEE II method. In order to get a more insight on the problem 
at hand, we suggest to the Dean to provide other information (s)he is sure about. For this reason, the Dean 
states that, locally, S2 is preferred to sq and ss is preferred to s±. 

Translating these preference information using the constraints Ch B (P B (2, G),fi) > and Ch B (P B (8, 1), (1) > 
0, and computing again the necessary and possible preference relations locally and considering both the 
bipolar PROMETHEE methods, we get the results shown in Tables 2] and [5j In these Tables, yellow cells 
correspond to new information we have got using the second piece of information provided by the Dean. 



Table 4: Necessary preference relations after the second piece of preference information 
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Table 5: Possible preference relations after the second piece of preference information 



(a) Local (b) PROMETHEE II (c) PROMETHEE I 
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In particular, in Tables 0] the cell in correspondence of the pair of students (si,Sj) is yellow colored if Sj 
was not necessarily preferred to Sj after the first iteration, but Si is necessarily preferred to Sj after the 
second iteration; in Tables O the cell in correspondence of the pair of students (sj, Sj) is yellow colored if Sj 
was possibly preferred to Sj after the first iteration but Sj is not possibly preferred to Sj after the second 
iteration anymore. Looking at Tables |4] and [5j the Dean is addressed to consider S7 as the best student. In 
fact, also if S7 and S3 are locally necessarily preferred to all other six considered students, S7 is still the only 
one being necessarily preferred to someone else considering the bipolar PROMETHEE I method. Besides, 
looking at Tables O we get that S3 is the only student being possibly preferred to S7 locally and with respect 
to PROMETHEE I and PROMETHEE II but, at the same time, everyone except S4, is possibly preferred 
to S3 considering the bipolar PROMETHEE I method while four students (S5, sq, sj and ss) are possibly 
preferred to S3 with respect to the bipolar PROMETHEE I method. 



7 Conclusions 

In this paper we proposed a generalization of the classical PROMETHEE methods. A basic assumption 
of PROMETHEE methods is the independence between criteria which implies that no interaction between 
criteria is considered. In this paper we developed a methodology permitting to take into account interaction 
between criteria (synergy, redundancy and antagonism effects) within PROMETHEE method by using the 
bipolar Choquet integral. In this way we obtained a new method called the Bipolar PROMETHEE method. 



The Decision Maker (DM) can give directly the preferential parameters of the method; however, due to their 
great number, it is advisable using some indirect procedure to elicit the preferential parameters from some 
preference information provided by the DM. 

Since, in general, there is more than one set of parameters compatible with these preference information, 
we proposed to use the Robust Ordinal Regression (ROR) to consider the whole family of compatible sets 
of preferential parameters. We believe that the proposed methodology can be successfully applied in many 
real world problems where interacting criteria have to be considered; besides, in a companion paper, we 
propose to apply the SMAA methodology to the classical and to the bipolar PROMETHEE methods (for a 
survey on SMAA methods see |21j). 
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